Using edit distance to analyse errors in a natural language to logic translation corpus

We have assembled a large corpus of student submissions to an automatic grading system, where the subject matter involves the translation of natural language sentences into propositional logic. Of the 2.3 million translation instances in the corpus, 286,000 (approximately 12%) are categorized as being in error. We want to understand the nature of the errors that students make, so that we can develop tools and supporting infrastructure that help students with the problems that these errors represent. With this aim in mind, this paper describes an analysis of a significant proportion of the data, using edit distance between incorrect answers and their corresponding correct solutions, and the associated edit sequences, as a means of organising the data and detecting categories of errors. We demonstrate that a large proportion of errors can be accounted for by means of a small number of relatively simple error types, and that the method draws attention to interesting phenomena in the data set

Proofs and Pictures Proving the Diamond Lemma with the grover Theorem Proving System

In this paper we describe a theorem proving system called grover. grover is novel in that it may be guided in its search for a proof by information contained in a diagram. There are two parts to the system: the underlying theorem prover, called &, and the graphical subsystem which examines the diagram and makes calls to the underlying prover on the basis of the information found there. We have used grover to prove the Diamond Lemma, a non-trivial theorem from the theory of well-founded relations. Key words. Automated reasoning, graphical theorem proving, proof strategies. This material is based upon work supported by the National Science Foundation under award number ISI-8701133. 1 INTRODUCTION 2 1 Introduction Open almost any mathematics text book and you will find, along with the familiar symbolism of mathematics and motivational text, many diagrams which are included to help the reader visualize the particular point being made. One might be tempted to conclude that mathema..

Architectures for Heterogeneous Reasoning on Interlinguae

The design of any computer system with a multimedia interface involves the designer in fundamental theoretical questions concerning the manipulation of information expressed in a variety of forms. Furthermore, if the system is to be intelligent then it must reason with and about the information that it represents. We use the term heterogeneous reasoning to refer to this task of reasoning with information presented in multiple forms. We define a heterogeneous reasoning system (HRS) as a composite deductive system which includes multiple component subsystems, each with its own syntax, semantics, and proof theory, and which also includes deductive rules which operate between the different subsystems. In the first half of this paper we provide a formal model which is able to distinguish several important types of heterogeneous reasoning. Computer systems designers whose tasks involve the use of an HRS in an intelligent multimedia interface are typically faced with a choice of several impl..

Student translations of natural language into logic : the grade grinder translation corpus release 1.0

A key characteristic of many existing referring expression generation (REG) algorithms is serial dependency: attributes are selected for inclusion one at a time, and the decision to include each attribute is dependent on the discriminatory ability of the set of attributes that have already been selected so far. We use a machine learning approach to explore whether serial dependency is a characteristic of human referring expression generation. Our results show that models in which attributes are chosen in a serially dependent fashion does not perform better than one where their inclusion depends only on other factors. The results also suggest that the visual salience of an attribute might be more important than its discriminatory power.10 page(s

The Role of Diagrams in Mathematical Proofs

. This paper describes our research into the way in which diagrams convey mathematical meaning. Through the development of an automated reasoning system, called grover, we have tried to discover how a diagram can convey the meaning of a proof. grover is a theorem proving system that interprets diagrams as proof strategies. The diagrams are similar to those that a mathematician would draw informally when communicating the ideas of a proof. We have applied grover to obtain automatic proofs of three theorems that are beyond the reach of existing theorem proving systems operating without such guidance. In the process, we have discovered some patterns in the way diagrams are used to convey mathematical reasoning strategies. Those patterns, and the ways in which grover takes advantage of them to prove theorems, are the focus of this paper. Key words: Mathematical diagrams, reasoning strategies, visualization, proof, automated reasoning. 1. Introduction Open almost any mathematics text bo..

Diagrams 2006 - Fourth International Conference on the Theory and Applications of Diagrams (Lecture Notes in Computer Science 4045, Springer)

Diagrams 2006 is the fourth event in this conference series, which was launched in Edinburgh in September 2000, and which has emerged as the major international conference on this topic. For Diagrams 2006 we especially solicited papers in the area of Diagrams and Education - a topic that includes uses of diagrams in all subject areas and phases of education from primary schools to professional development, and which also concerns education for diagrammatic literacy. The call for papers for Diagrams 2006 solicited contributions of full papers, extended abstracts and tutorial proposals. Submissions were received from authors representing both academia and industry, twenty-three countries, % affiliation countries in submissions ()=double affiliation % Australia Austria Brazil Canada China France Germany Hungary India % Israel Italy Japan Korea Mexico Netherlands New_Zealand Norway % (Philippines) Poland (Spain) Taiwan UK USA and disciplines including: % Computer Science % Engineering % Division of Interactive and Intelligent Computing\\\\\\\\\\\\\\\\% NASA Ames Research Center % Department of Media Studies % Department of Spatial Information Science and Engineering \\\\\\\\\\\\\\\\% National Center for Geographic Information and Analysis\\\\\\\\\\\\\\\\% Department of Mathematical Sciences % Department of Informatics % Department of Education % Department of Educational Psychology % The Faculty of Physics, Astronomy and Applied Computer Science % Telematics % Department of Mathematics and Informatics % Department of Psychology \\\\\\\\\\\\\\\\% Engineering Design Centre % Faculty of Culture and Information Science\\\\\\\\\\\\\\\\% Naval Research Laboratory % Informatics % Psychology and Human Development % Philosophy % Department of Humanities Computing % Research group on Mathematical Linguistics % Graduate School of Computational Design % Graduate school of Information, Productions and Systems % IBM % Information Technology % Amazon aerospace,computer and information science, education, engineering, geographical and spacial information science linguistics, mathematics, philosophy, psychology and telecommunications. Submissions were reviewed by a distinguished international program committee with members from ten countries. Each submission was peer-reviewed by three members of the program committee or by reviewers that they nominated. This labor-intensive process was designed to ensure that only the submissions with the greatest technical merit, clearest communication, and widest interest were included in the conference. By this process we sought to maintain the quality and multidisciplinary balance characteristic of the conference series. Keeping to the high standards of previous Diagrams conferences, the acceptance rate for full papers was about 30\\\\\\\\\\\\\\\\% (thirteen papers), and that for extended abstracts 60\\\\\\\\\\\\\\\\% (twenty-two abstracts). Nine submissions chosen from the best of the extended abstract

Impedance Effects of Visual and Spatial Content upon Language-to-Logic Translation Accuracy

There is a body of work that suggests that those elements of the cognitive architecture responsible for processing, on the one hand, visual information (essentially visual properties of objects), and, on the other hand, spatial information (spatial relationships between objects), may compete with each other for resources. In this paper, we explore whether and to what degree the processing of visual and spatial information interferes with the task of translation from natural language into logic, a skill that students often find difficult to master. Using a large corpus of student data, we determine correlations between difficulty and the particular properties used in the sentences, with implications for pedagogical design